add Apache Comet #557
add Apache Comet #557
Conversation
|
|
| @@ -0,0 +1,236 @@ | |||
| { | |||
There was a problem hiding this comment.
These machine types were measured for Spark: c6a.2xlarge.json c6a.4xlarge.json c6a.xlarge.json c8g.4xlarge.json
Do you plan to add them as well or do you like me to do this?
There was a problem hiding this comment.
I can probably run that too after we finalize discussion on the PR.
This comment was marked as resolved.
This comment was marked as resolved.
|
@rschu1ze Currently I'm solving AWS issues related to my account and I don't know how long this will take. If you have an opportunity to run bench yourself I'd appreciate it. Otherwise I'll run it myself as soon as possible. |
|
The PR LGTM. It will be hard for me to capture measurements on c6a.2xlarge.json, c6a.4xlarge.json, c6a.xlarge.json, and c8g.4xlarge.json in the next 1.5 weeks but if you find some time, please go ahead. Otherwise or if this is urgent, I can also merge this PR (just lmk). |
|
@rschu1ze, we have the automation - please go ahead and merge. Then we can run on any machines in a matter of a single command. |
|
@alexey-milovidov current VM's (c6a.4xlarge) results should be re-run too since last changes. |
|
No problem, the most important part is that the scripts are ready. Thank you! |
|
Good news, the script runs well, both on amd64 and aarch64. |
|
Except for the small machine, t3a.small: |
This PR adds Apache Comet as a backend for Spark in analytical workloads. Base issue - #517.
Notes:
Also I consider creating issue in Comet repo for devs to double-check my configuration. This is especially important since there are 1 query that failed and other questionable moments. Meanwhile, I'd glad to hear comments from interested.